Comparison of Current and Historical Ingestions

When you ingest current data (either manually or through a scheduled ingestion), the data ingestion and snapshot creation process consists of ingesting from a live production data source. The resulting snapshot is identified by a time stamp with the date of the ingestion. This identifies when the data snapshot was current. These snapshots can be used by other applications to represent what the data looked like at a specific point in time.

In order to report on data from the past which is not represented by an existing snapshot, you can use the historical ingestion functionality. To do this, you must have a data source that contains the legacy data from the date you need. You then can build the data mapping for that data which in turn, allows you to ingest the data into DORIS.

Historical data ingestions and snapshots are especially important when dealing with data sources that have frequent updates. When you use a frequently-changing data source for a date-critical report, a current ingestion may not reflect values as they existed on the actual reporting date. This is why you should take a snapshot of the data on the reporting date to eliminate the chance of capturing data that was updated after the report date.

Applications such as IRIS prioritize use of certain snapshots based on relevance of the snapshot date against the intended reporting range.

 

See also:

• Ingesting the Data
• Scheduling Data Ingestions for Current Data
• Typical Process Workflow
• Historical Process Workflow